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ABSTRACT 

The purpose of this paper is to determine whether the changes that were found to occur pre- to post intervention in 
students' cognitive structures (Mills, 2013; Knezek, Christensen, Tyler-Wood, & Periathiruvadi, 2013) continued to 
persist two years later. Major findings were: a) semantic perception of science and STEM as a career became more 
aligned with interest in being a scientist, from pretest to post test time during the treatment year and continued to be 
aligned two years later; b) semantic perception of engineering moved from alignment with science and STEM as a career 
at time 1, to alignment with semantic perception of technology and creative tendencies after the treatment year, at time 2, 
and remained aligned with technology two years later, at time 3; and c) semantic perception of mathematics was 
separated from the other constructs during the pre-post treatment year and remained largely separated two years later. 
Data mining techniques were also used to explore changes in relationships among these and other constructs over time. 
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1. INTRODUCTION 

Middle school students were the target of this hands-on energy monitoring project. The Middle Schoolers 
Out to Save the World (MSOSW) project was funded from the Innovative Technology Experiences for 
Students and Teachers (ITEST) program of the National Science Foundation (NSF). The main goal of the 
MSOSW project has been to interest and prepare middle school students to participate in the science, 
technology, engineering and mathematics (STEM) workforce of the future. 

While there were 15 classrooms in eight schools participating in the MSOSW project, the researchers 
were able to gather longitudinal data two years after the treatment for one school (with three classrooms). 
These students were surveyed three times, once prior to the treatment (pretest) in the fall of their sixth grade 
year, once following the treatment (post test) in the spring of their sixth grade year and again two years later 
in the spring of their 8 th grade year. 


2. RESEARCH STUDY 
2.1 Instrumentation 

Three different survey instruments were used to gather project data. The data were gathered online via the 
project website. The STEM Semantics Survey (Tyler-Wood, Knezek, & Christensen, 2000) was used to 
measure perceptions of science, mathematics, engineering, technology, and STEM careers, while the 
Computer Attitude Questionnaire (CAQ) (Knezek & Christensen, 1998) was used to measure self-perceived 
motivation, creative tendencies, and attitudes toward school. The Career Interest Questionnaire (CIQ) 
(Bowdich, 2009) was used to examine student attitudes toward a career in science. 

The STEM Semantic Survey is a semantic differential instrument designed to assess elementary through 
secondary students’ as well as pre-service and in-service teachers’ perceptions of STEM disciplines. 
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This instrument consists of 25 items, divided into 5 sub-scales: Science, Mathematics, Engineering and 
STEM as a Career. The items for each of the five scales are semantic adjective pairs such as 
boring/interesting or exciting/unexciting. Previous studies using this instrument (n = 174) revealed internal 
consistency reliabilities on perceptions of science, math, engineering, technology, and STEM as a career 
ranged from a = .84 to a = .93 (Tyler- Wood, Knezek, & Christensen, 2010). 

Learner disposition measurement scales from the Computer Attitude Questionnaire (Self concept, 
Creative Tendencies, and Attitudes toward School are also utilized in this study (Knezek, Christensen, 
Miyashita & Ropp, 2000). These scales are comprised of Likert-type question items with response ratings 
ranging from strongly disagree (1) to strongly agree (5). The reliabilities of these scales ranged from a = .72 
to a = .88 in a prior study (Knezek & Christensen, 2000). 

The CIQ is a Likert-type (1 = strongly disagree to 5 = strongly agree) instrument composed of 12 items in 
three sections that can each form a measurement scale. Both the 3 parts as well as the 12-item total CIQ scale 
was used in this study. The CIQ is adapted from an instrument developed a project promoting STEM interest 
(Bowdich, 2009). Reliabilities for the CIQ total scale score based on 12 items have typically fallen in the 
range of alpha = .94, while part a, b, and c reliabilities have typically ranged from .78 to .94 (Tyler-Wood, 
Knezek, & Christensen, 2010). 

2.2 Methods 

Follow up data were collected from one set of students (n=60) two years after they participated in the 
MSOSW project activities as sixth graders. These students completed pretest surveys in fall of 2010 (time 1), 
posttest surveys in spring of 201 1 (time 2) and follow up surveys in spring 2013 (time 3). 

2.3 Findings 

2.3.1 Descriptive Findings 

Descriptive data of the significant changes in dispositions are supplied in Table 1 for all three administrations 
of the surveys. As shown in Table 1, this group of students generally followed the trend noted in previous 
studies in steady declines in learning dispositions as students progress through grade levels (and age) in 
school (Knezek & Christensen, 2000; Christensen & Knezek, 2001). One noteworthy exception is in Attitude 
Toward School that exhibited a significant increase from pretest to post test time during the MSOSW 
activities of their sixth grade year, before reverting to the general trend towards decline by the eighth grade 
year. 


Table 1. Analysis of Variance for Three Administrations of Survey Instruments 



N 

Mean 

Std. Dev. 

Sig. 

CAQ Attitude Toward 
School 

Time 1 

63 

3.06 

.64 


Time 2 

53 

3.18 

.51 


Time 3 

60 

2.90 

.60 


Total 

176 

3.04 

.60 

.039 

CAQ Self Concept 

Time 1 

63 

4.83 

1.05 


Time 2 

53 

3.96 

.63 


Time 3 

60 

3.85 

.66 


Total 

176 

4.23 

.92 

.000 

STEM Science 

Time 1 

63 

5.98 

1.11 


Time 2 

53 

4.92 

1.37 


Time 3 

59 

4.98 

1.52 


Total 

175 

5.32 

1.42 

.000 

STEM Technology 

Time 1 

63 

6.27 

1.10 


Time 2 

53 

5.89 

1.20 


Time 3 

59 

5.73 

1.41 


Total 

175 

5.97 

1.26 

.048 

CIQ Parti 

Time 1 

63 

3.21 

.79 


Time 2 

53 

2.85 

.97 


Time 3 

59 

3.12 

.94 


Total 

175 

3.07 

.90 

.087 
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2.3.2 Gender Differences 

Gender differences for the key measures exhibiting significant (p < .05) differences at time 1 are shown in 
Table 2. As shown in Table 2, at the beginning of the sixth grade year, before taking part in MSOSW 
activities, girls reported significantly (p < .05) lower semantic perceptions of science, engineering, and 
STEM as a career than boys. Effect sizes for boys versus girls ranged from .46 for STEM as a Career to 1.06 
for perception of engineering. These lie in the range of moderate to large according to guidelines by Cohen 
(1988). They surpass the ES = .3 criteria at which the magnitude of a difference is normally considered 
educationally meaningful (Bialo & Sivin-Kachala, 1996). 

After completion of MSOSW activities, at the end of the sixth grade year in school, a very different 
picture regarding gender differences emerged. Specifically, there were no (p < .05) significant differences 
between boys and girls on any of the three measures of science, engineering, and STEM as a career. In 
addition, effect sizes had been reduced to the point at which none of the three identified measures indicated 
educationally meaningful (ES > .3) differences between the semantic perceptions of science, engineering, 
and STEM as a career for boys and girls. As shown in Figure 1, generally the boys declined a large amount 
from pretest to posttest, while girls declined little or (in the case of engineering) became more positive in 
their perceptions from pretest time to post. This implies that the MSOSW activities had an especially positive 
impact on girls, as was reported for the overall findings across schools by Knezek, Christensen, Tyler -Wood, 
and Periathiruvadi (2013). 

At the time of the follow-up assessment two years later, at the end of the eighth grade year (time 3), girls 
had generally retained their end-of-sixth grade perceptions of science, engineering, and STEM as career 
(Figure 1), but the boys had rebounded to the point where their dispositions toward science, technology and 
engineering were once again significantly (p < .05) more positive than girls in all three areas of semantic 
perception of science, engineering, and STEM as a career. 

Table 2. Significant Differences between Males and Females on STEM Measurement Indices 



Gender 

n 

Mean 

SD 

Sig 

ES 

STEM Science Time 1 

Male 

31 

6.30 

.95 



Female 

32 

5.67 

1.19 



Total 

63 

5.98 

1.11 

.024 

.59 

STEM Engineering Time 1 

Male 

31 

5.91 

1.49 



Female 

32 

4.30 

1.56 



Total 

63 

5.09 

1.72 

.000 

1.06 

STEM Career Time 1 

Male 

31 

5.62 

1.22 



Female 

32 

4.94 

1.68 



Total 

63 

5.27 

1.50 

.069 

.46 








STEM Science Time 2 

Male 

25 

5.00 

1.54 



Female 

28 

4.86 

1.24 



Total 

53 

4.92 

1.37 

.709 

.10 

STEM Engineering Time 2 

Male 

25 

5.18 

1.93 



Female 

28 

4.71 

1.37 



Total 

53 

4.93 

1.66 

.303 

.28 

STEM Career Time 2 

Male 

25 

4.64 

1.69 



Female 

28 

4.39 

1.42 



Total 

53 

4.51 

1.54 

.548 

.16 








STEM Science Time 3 

Male 

33 

5.32 

1.58 



Female 

26 

4.54 

1.33 



Total 

59 

4.98 

1.52 

.048 

.54 

STEM Engineering Time 3 

Male 

32 

5.01 

.81 



Female 

26 

4.10 

.97 



Total 

58 

4.60 

.99 

.000 

1.02 

STEMCareer5 time 3 

Male 

33 

5.40 

1.58 



Female 

27 

4.55 

1.16 



Total 

60 

5.02 

1.46 

.023 

.61 
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Figure 1. Changes Pre-Post for these Three Across Time 1, 2, 3. 

2.3.3 Correlational Relationships for Time 1, 2 and 3 

Trends in correlations that changed from time 1, to time 2 and time 3 were a primary impetus for examining 
these data from a construct-based perspective. As shown in Table 3, semantic perception of technology was 
the only STEM measure strongly associated with other gathered indicators at the time of the pretest, time 1 . 
Especially notable is the positive association between STEM technology measures and creative tendencies (r 
= .303, p < .016). By the end of the sixth grade treatment year, significant positive correlations existed 
between creative tendencies and semantic perception of engineering (r = .377, p < .005) semantic perception 
of technology (r = .275, p < .047), CIQ Part 1 (r = .275, p < .047), CIQ Part 2 (r = .327, .017), CIQ Part 3 (r 
= .285, p = .040), and the CIQ total scale score (r = .346, p < .011). Associations had developed over the 
course of the year of MSOSW activities between creative tendencies and numerous measures of interest in 
“doing science” or “being a scientist” as a career. 

Two years later, at time 3, creative tendencies had retained its significant (p < .05) associations with CIQ 
Part 1 (family and environmental influence on STEM career) (r = .263, p < .044) and on science in the form 
of semantic perception of science (r = .310, p < .017). In addition, computer enjoyment had become aligned 
with semantic perception of STEM as a career (r = .331, p < .01). These patterns indicate that the cognitive 
perceptions of the MSOSW participants underwent an extensive change from the beginning to the end of the 
MSOSW treatment activities school year. In addition, evidence shows that some of these were still retained 
two years later. 

Systematic examination of changes in these inter-correlations led to exploration of the emergence and 
retention of these constructs through higher-order factor analysis and data mining techniques. 

2.3.4 Higher Order Factor Analysis 

Higher order factor analysis is a type of exploratory factor analysis in which the variables to be factor 
analyzed are scale scores rather than individual item responses (Dunn-Rankin, Knezek, Wallace, & Zhang, 
2004). 

Each scale score typically itself represents a defined construct (such as creative tendencies or perception 
of mathematics) and therefore the rationale for the process is searching for higher-order constructs that may 
explain an individual’s attributes on several constructs related at a higher level. 

As shown in Table 3, at the time of the pretest when the students were beginning the sixth grade, the three 
parts of the CIQ were aligned in Factor 1 together while semantic perception of engineering and semantic 
perception of STEM as a career and semantic perception of science were together in Factor 2. Factor 3 
contained creative tendencies and semantic perception of technology, while in Factor 4 mathematics emerged 
as a higher order construct on its own. The four factors extracted explained 77.0% of the common variance in 
the data. Note that semantic perceptions of science and STEM as a career were not strongly aligned with the 
CIQ scales relevant to having a career as a scientist or doing science, and creative tendencies clustered with 
semantic perceptions of technology. 
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Table 3. STEM-related Constructs Existing Among 6 th Grade Students as of Pretest Time of the Treatment Year 



Component ( 


1 

2 

3 

4 

CIQ Part 3 

.847 



.264 

CIQ Part 1 

.816 

.411 


.135 

CIQ Part 2 

.800 

.406 


-.142 

STEM Engineering 

.306 

.781 



STEM Career 

.437 

.648 


.354 

STEM Science 

.381 

.421 

.287 

.382 

CAQ Creative Tendencies 

.115 

-.363 

.804 


STEM Technology 


.387 

.789 


STEM Mathematics 




.942 


Notes: Extraction Method: Principal Component Analysis, Rotation Method: Varimax with Kaiser Normalization, Rotation 
converged in 4 iterations, Factor loadings < .1 suppressed 


By the end of the sixth grade year, CIQ Part 1 (family and environmental support for career as scientist) 
and Part 2 (interest in college courses in science) clustered together in Factor 1 with semantic perception of 
STEM as a career and semantic perception of science. Semantic perception of engineering joined creative 
tendencies and semantic perception of technology in Factor 2. CIQ Part 3 (making the world a better place 
through science) separated out to be on its own higher order construct, and semantic perception of 
mathematics continued as its own higher order construct, in Factors 3 and 4. The four factors extracted 
explained 75.1% of the common variance in the data. These relationships are shown in Table 4. 

Table 4. STEM-related Constructs Emerging Among 6 th Graders by the end of Treatment Year (time 2) 



Component | 


1 

2 

3 

4 

CIQ Part 2 

.852 


.189 

-.138 

CIQ Part 1 

.800 

-.101 

.434 

-.143 

STEM Career 

.595 

.568 

-.182 


STEM Science 

.564 

.219 

.512 

.361 

STEM Engineering 

.227 

.774 



STEM Technology 

-.323 

.766 

.230 


CAQ Creative Tendencies 

.219 

.543 


-.537 

CIQ Part 3 

.239 

.133 

.868 

-.165 

STEM Mathematics 




.852 


Notes: Extraction Method: Principal Component Analysis, Rotation Method: Varimax with Kaiser Normalization, Rotation com 
in 7 iterations, Factor loadings < . 1 suppressed 


As shown in Table 6, at the time of the follow up survey two years later, CIQ parts 1 (family and 
environmental support for career as scientist) and Part 2 (interest in college courses in science) remained 
clustered together in Factor 1 with semantic perception of STEM as a career and semantic perception of 
science, while semantic perception of engineering and semantic perception of technology remained clustered 
in Factor 2. CIQ Part 3 (making the world a better place through science) also remained as its own higher 
order construct in Factor 3, while creative tendencies separated from its time 2 clustering with semantic 
perception of engineering and semantic perception of technology in Factor 2 and became the opposite pole of 
semantic perception of mathematics in Factors 4. The four factors extracted explained 78.1% of the common 
variance in the data. These relationships are shown in Table 5. 

Table 5. STEM-related Constructs Existing Among 6 th Graders at the end of their 8 th Grade Year 



Component | 


1 

2 

3 

4 

CIQ Part 2 

.932 




CIQ Part 1 

.870 

.117 

.301 

.103 

STEM Career 

.674 

.528 

-.169 


STEM Science 

.571 

.488 

.321 

.304 

STEM Technology 


.849 

.313 


STEM Engineering 

.366 

.756 
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CIQ Part 3 

.157 

.152 

.928 


CAQ Creative Tendencies 


.230 


.824 

STEM Mathematics 


.439 

-.145 

-.607 


Note: Extraction Method: Principal Component Analysis, Rotation Method: Varimax with Kaiser Normalization, Rotation converged 
in 7 iterations, Factor loadings < . 1 suppressed 


2.3.5 Evolution of Constructs over Time Based on Higher Order Factor Analysis 

Several changes in the conceptual frameworks of the students involved in this project can be identified as still 
persisting at the time of the two-year follow-up study. These can be identified through higher-order factor 
analysis even as the results of ANOVAs comparing increases or decreases at time 1 versus time 2 and time 3 
are inconclusive at best. The major trends can be summarized as: 

1. Desire to be a scientist (CIQ Parts 1, 2, 3) formed its own factor prior to initiation of MSOSW 
activities at the beginning of the sixth grade school year. Desire to be a scientist and the semantic perception 
of a career in STEM became attached to semantic perception of science by the end of the sixth grade 
treatment year. This alignment of desire to be a scientist with semantic perception of science continued to 
persist two years later, when the students were at the end of their eighth grade year in school. 

2. Semantic perception of science was clustered together with semantic perception of engineering (in 
Factor 2) at the beginning of the sixth grade year for these students. By the end of the sixth grade treatment 
year semantic perception of science had moved to be clustered with desire to be a scientist (CIQ Parts 1 and 
2), while semantic perception of engineering, technology, and creative tendencies clustered together to form 
Factor 3. Two years later, at the end of the eighth grade year, semantic perception of STEM as a career 
remained aligned with semantic perception of science and semantic perception of STEM career. 

3. Semantic perception of mathematics, at the time of the pretest and posttest of the sixth grade year, was 
its own Factor 4. By the end of the eighth grade year, semantic perception of mathematics was joined by 
creative tendencies but as a polar opposite meaning those who had higher perceptions of creative tendencies 
had lower semantic perceptions of mathematics. 

2.3.6 Analysis Based on Data Mining Techniques 

Using the Eureqa data mining software developed at Cornell University, three measures were compared: 
Time 1, 2 and 3 which represent a pre-assessment, a one year post and a two-year post. Searches of 1 min 30 
sec were performed on each data set, during which time approximately 2.2 * 10 A 9 evaluations were 
performed comparing potential symbolic expressions against a metric of minimizing the absolute error 
(Figure 2). A progress over time graph for each search represented the drop in absolute error over time and 
helped verify that a reasonable level of stability (e.g. a horizontal line) had been reached at the time of the 
conclusion of the search. 


Run the formula search: 


Run 


Pause 


Stop 


Progress and performance 


Progress over time 


Time 

Search time 

lm 30s 

CPU cores 

0 

Performance 

Generations 

27589 

Generations/sec 

325.1 

Formula evaluati... 

2.2e9 

Evaluations/sec 

25.90 M 

Confidence 

Stability 

30.5% 

Maturity 

21.5% 



Figure 2. Graphic representation of a search for equations given a data set documents the time of the search, the 
performance of the computational method, and confidence metrics. 

At the conclusion of each search, a decision is made concerning which equation to select to represent the 
best trade-off of complexity versus error with an acceptable correlation coefficient. A number of comparisons 
aid in the decision, for example, the gain in correlation coefficient per unit of complexity, the change in 
number of coefficients, and the use of variables within the candidate expressions. In general, a solution is 
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selected that minimizes both error and complexity and which has an interpretable mathematical structure. 
The analysis then proceeds to find and interpret changes among the selected equations that may have 
statistical support from regression analysis, that may have external support from previous research, that may 
help further explain or validate previous results, and that may suggest additional analyses. 


Best Solutions of Different Sizes: 


Solution Fit Plot (default) 


yj CIQTotal = 3.56 - 0.758 sin(0.5 1 6 STEMScienct 
| CIQTotal = 3.65 + 0.0478 STEMScience STEM! 
| CIQTotal = 4.44 + 0.344 STEMScience + 0.082f 



1 CIQTotal = 1 .6 + 0.7 1 8 CAQAttitudeTowardSchi 

19 U2J CIQTotal = 1.4 + CAQAttitudeTowardSchool + i 
21 CIQTotal =1.85 CAQAttitudeTowardSchool + 0 

Solution Details: 

Solution 

CIQTotal = 1.602 + 

0.7 1 76‘CAQAttitudeTowardSchool + 0.04706*ST... 

RA2 

Goodness 

0.4809928 

Correlation 

Coefficient 

0.70077893 

Maximum 

Error 

1.69777S5 

Mean 

Squared Er ... 

0.25564664 

Mean 

Absolute E... 

0.36574502 


CIQTotal(train) 
CIQT otal(val Idatlon) 





0 

10 20 30 

40 

50 

60 

<row> 




F 


Solutions Plotted Accuracy v 

s Complexity 




0.46 
.0.44 
t 0.42 
0.4 
0.38 
0.36 
0.34 


Frontier 
Solutions 
size 17 • 


Figure 3. Equation options at the conclusion of a search showing the position (red dot in the lower right panel of the 
graphic) of the selection along the Pareto curve of complexity versus accuracy. Full solution details are reported in the 

narrative. 


Table 6. Results of Search for Best Predictors of Science Career Interest Using Eureqa Data Mining Software 


CIQTotal = f(CAQCreativeTendencies, CAQAttitudeTowardSchool, CAQSelfConcept, STEMScience, 
STEMMathematics, STEMEngineering, STEMTechnology) 


Time 1 

Time 2 

Time 3 

R A 2 Goodness of Fit 

0.52122356 

0.52367202 

0.58247764 

Correlation Coefficient 

0.72217347 

0.72538579 

0.76433976 

Complexity 

19 

21 

21 

Time 1 Solution 

CIQTotal = CAQAttitudeTowardSchool + 0.08586*CAQCreativeTendencies * 
STEMMathematics + 0.02497 *STEMScience*STEMEngineering - 
0.1384*CAQAttitudeTowardSchool * STEMMathematics 

Time 2 Solution 

CIQTotal = 3.836*CreativeTendencies + 0.1653*CAQCreativeTendencies*STEMScience - 
4.253 - 0.3318*STEMScience - 0.5892*CAQCreativeTendencies A 2 

Times 3 Solution 

CIQTotal = 1.789*CAQAttitudeTowardSchool + 0.3188*STEMScience + 
0.06987*STEMTechnology A 2 - 1.364 - 
0.277*CAQAttitudeTowardSchool*STEMTechnology 


Major contributions of the Eureqa data mining analyses were: a) to reconfirm that semantic perception of 
science plays an important role in predicting reported interest in Being a Scientist ; b) to reconfirm the 
important association of creative tendencies at time 1 and time 2 with interest in Being a Scientist , well as to 
highlight questions about why creative tendencies became disconnected (not a major predictor) by the end of 
the eighth grade year, at time 3; and c) to note that attitude toward school was a major predictor of interest in 
Being a Scientist at time 1, was not a major predictor at time 2, and then returned to be a major predictor at 
time 3. Eureqa findings collectively imply that the hands science activities introduced during the sixth grade 
may have ameliorated the well-known trends toward decline in attitudes toward school as students advance 
through higher grade levels. Further research is needed in this area. 


3. CONCLUSION 

Semantic perception of Science and of STEM as a Career became more closely aligned with Doing Science 
and Being a Scientist over the course of the MSOSW grant activity year. This alignment remained in place 
two years later. Two major outcomes of the MSOSW project were related to the current study. 
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First, students emerge from project activities with increased aspirations for STEM -related careers with 
gains in students’ understanding of what scientists do, and with gains in belief that scientists can work on 
things that help the world. Second, female students gained more than males in their perceptions of science, 
engineering, mathematics and technology (Knezek, Christensen, Tyler-Wood & Periathiruvadi, 2013). The 
current findings provide further evidence that the positive impact of MSOSW persists in student participants 
for several years beyond the time frame of the project. This finding is consistent with a nine-year follow-up 
study of BUGS girls (Tyler-Wood et al, 2011) that used the same instruments employed in the current study 
to demonstrate that participants’ higher STEM dispositions (vs. childhood comparison group) persisted from 
elementary school into university studies. 
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